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Abstract. In this paper, we consider a multiobjective optimal control problem where the 
preference relation in the objective space is defined in terms of a pointed convex cone containing the 
origin, which defines generalized Pareto optimality. For this problem, we introduce the set-valued 
return function V and provide a unique characterization for V in terms of contingent derivative and 
coderivative for set- valued maps, which extends two previously introduced notions of generalized 
solution to the Hamilton-Jacobi equation for single objective optimal control problems. 
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, 1. Introduction. Many engineering applications, such as trajectory planning 

^> ' for spacecraft [5] and robotic manipulators [5], continuous casting of steel [S], etc., 

can lead to an optimal control formulation where p objective functions (p > 1) need 

' to be optimized simultaneously. For multiobjective optimization problems, optimality 

is determined by the preference of the decision maker, which is expressed in terms 
of a binary relation. In this paper, we will consider the preference defined in terms 
i—i, of a pointed convex cone P C R p containing the origin [19] , which, when P — R^_, 

yields the well-known Pareto optimality. The derivation of optimality conditions for 
multiobjective optimal control problems (MOC) with more general preferences than 

■ the one considered in this paper has been a subject of recent interest. Mostly, the 

, approach [3J [T^l [20] has been to derive necessary optimality conditions. In this paper, 

we take a different approach, treating (MOC) in the framework of dynamic program- 
ming [2]. Namely, for an autonomous multiobjective optimal control problem with 
fixed end time where the dynamics are governed by a differential equation, we first 
define a set-valued return function V. Using the concepts of contingent derivative [TJ 
p. 181] and coderivative [H] for set-valued maps, we then provide a unique characteri- 
. . , zation for V, which extends the notions of generalized solution to the Hamilton-Jacobi 

equation for single objective optimal control problems introduced in [181 p. 454]. 

X : 

' A work related to ours is 0], where the same multiobjective optimal control 



problem is considered, but the preference is given by the lexicographical order. In 
U, a "vector Value function" for the problem is first defined. Two notions of so- 
lution to a suitable system of Hamilton-Jacobi equations are then introduced: the 
notion of contingent solution, which uses the concept of contingent vector epideriva- 
tive, and the notion of vector viscosity solution. The main result in j4] is to show 
that the "vector Value function" is the unique vector extended lower semicontinuous 
contingent and vector viscosity solution. This is also the program followed in this 
paper for (MOC). More precisely, we start by defining the set-valued return function 
V(-, •) : [0, T] x R™ — > 2 R as the set- valued map associating with each time t € [0, T] 
and state x e R™ the set of generalized Pareto optimal elements (which we call min- 
imal elements) in the objective space Y(t,x), where Y(t,x) is the set of all possible 
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values that can be taken by the vector-valued objective function for trajectories start- 
ing at x at time t. The set-valued return function V is an extremal element map, 
i.e., a map whose set values only contain minimal elements. For such maps, using 
the concepts of contingent derivative and coderivative for set-valued maps, we then 
extend the two notions of generalized solution to the Hamilton- Jacobi equation for 
single objective optimal control problems introduced in [18j p. 454]. We call these two 
extended notions generalized contingent solution and generalized proximal solution for 
(MOC). Our main result is to show that V is the unique set- valued map W general- 
ized contingent and proximal solution such that the set- valued map Wf := W + P is 
outer semicontinuous. The proof of this result is made of several steps. First, we show 
that V is a generalized contingent solution and that Vf is outer semicontinuous. In 
the process of this proof, we state the multiobjective dynamic programming equation 
for (MOC). Next, we show that generalized contingent solution implies generalized 
proximal solution. We finally obtain uniqueness for extremal element maps using in- 
variance theorems [15] . 

This paper is organized as follows. In <J2] we detail the class of multiobjective 
optimal control problems considered and define the set-valued return function V. In 

we then provide useful mathematical definitions. In SjU we discuss the concept 
of optimality in multiobjective optimization and present several properties of the 
minimal element set. Also, we provide a general sufficient condition for V to be a 
Lipschitz set-valued map. In <J5] we state the multiobjective dynamic programming 
equation for (MOC). In Sj5] we present the notion of generalized contingent solution. 
In SJ7] we prove that V is a generalized contingent solution and that Vf is outer 
semicontinuous. Finally, in <JS] we present the notion of generalized proximal solution, 
show that generalized contingent solution implies generalized proximal solution, and 
prove that V is the unique generalized proximal solution. 

2. A multiobjective finite- horizon optimal control problem (MOC) |2j. 

Consider the evolution over a fixed finite time interval I = [0,T] (0 < T < oo) of an 
autonomous dynamical system whose n-dimensional state dynamics are given by a 
continuous function f (•, •) : R" x U — > R", where the control space U is a nonempty 
compact subset of R m . The function f(-,u) is assumed to be Lipschitz, i.e., some 
Kf > obeys 



We also assume that the function f is uniformly bounded, i.e., some Mf > obeys 



A control u(-) : I — >• U is a bounded, Lebesgue measurable function. The set of 
controls is denoted by U. The continuity of f and the Lipschitz condition (|2.1|) guar- 
antee that, given any t £ I, initial state x £ R™, and control u(-) £ U, the system of 
differential equations governing the dynamical system, 



Vuef/, Vxi,x 3 eR n , ||f(xi,u) -f(x 2 ,u)|| <i*r f ||xi-x 2 | 



(2.1) 



Vx £ R n , Vu £ U, ||f(x,u)|| < M f . 



(2.2) 



{ 



*(*) 
x(t) 



f(x(s),u(s)), t<s<T, 



X 



(2.3) 



has a unique solution, called a trajectory and denoted s — > x(s; t, x, u(-)). Using the 
Gronwall inequality, the following estimate between trajectories can be obtained. 
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Proposition 2.1 (Estimate between trajectories). Let t e I, x l5 x 2 e R", and 
u(-) G U. Then, 

Vs e [t, T], ||x(s; t, xi, u(-)) - x(s; f, x 2 , u(-))|| < exp(K r (a - i))||xi - x 2 || . 

The cost of a trajectory over [t,T], t G I, is given by a p-dimensional vector 
function J(-, •,•,•): 7 X I X R™ xM-> R p , 

J(i, T, x, u(0) = f L(x( S ; x, u(-)), u(s)) ds, (2.4) 

where the p-dimensional vector function L(-,-) : R" x [/ — > R p is assumed to be 
continuous. For simplicity, no terminal cost is included in (|2.4[) . We assume that the 
function L is uniformly bounded, i.e., some A/l > obeys 

Vx G R™, Vu G U, ||L(x, u)|| < M L , (2.5) 

and that the function L(-, u) satisfies a Lipschitz condition, i.e., some Kl > obeys 

Vu G U, Vxi,x 2 G R", ||L(x 1 ,u)-L(x 2 ,u)|| < # L ||xi - x 2 ||. (2.6) 

Using (|2.6p and Proposition 12.11 the following estimate between the cost of two tra- 
jectories can be obtained. 

Proposition 2.2 (Estimate between costs). Let ti,t 2 G I, xi,x 2 G R™, and 
u(-) G U. Then, 

||J(ti,T,xi,u(-))- J(t 2 ,T,x a ,u(-))|| < -r^exp(iirfT)||xi-x a ||+Mi i |ti-t2|. 

Af 

The objective space F(i, x) for (MOC) is defined as the set of all possible costs (|2.4j) : 

y(t,x) = jj(t J T,x,u(-)),u(-) ewj. 

From (|2.5p . it follows that the set Y(t, x) is bounded (by MlT), however F(t, x) is not 
necessarily closed. Using Proposition 12.21 the following estimate between objective 
spaces can be obtained. 

Corollary 2.3 (Estimate between objective spaces). Lett\,t 2 <E I anrfxi,x 2 e 
R". Then, 

y(t x ,xi) cF(i 2 ,x 2 ) + ^exp(A'fT)||x 1 -x 2 ||+M L |t 1 -t 2 |^B, 

where B(x, £) is t/ie closed ball centered at x wzi/i radius I and B = B(0, 1). 

In ^4.31 we will need the norm || • || in R p to be Euclidian. Let (•,•) be the asso- 
ciated inner product. 

The set-valued return function V(-, •) : L x R" — > 2 RP for (MOC) is defined as the 
set- valued map which associates with each time t € I and initial state x G R™ the set 
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of minimal elements of the objective space Y(t,x) with respect to the ordering cone 
P, where the definition of a minimal element is postponed to i j4.ll 

V(i,x)=£(cl(Y(i,x)),P). (2.7) 

The closure in (j2.7[) is used to guarantee the existence of minimal elements (Proposi- 
tion 3.5, [7 ). Hence, V* G /, Vx € R™, V(t,x) ^ 0. 

Remark 2.1. When p = 1 and P — R+, (|2.7j) takes the form 

V(t,x) = ( inf f L(x(s;t,x,u(-)),u(s)) dsX. 

Hence, V(t,x.) = {v(t, x)}, where v(-, •) is i/ie wZwe function for single objective opti- 
mal control problems [21 118j . 

3. Mathematical preliminaries. In this section, we recall some useful defi- 
nitions and discuss general set-valued maps with values in R p . Let X be a normed 
linear space, F be a set-valued map from X to R p , K. — {S c R p , S ^ 0, S compact} 
and M = {ScRP, S^%, S bounded}. 

First, we provide the definition of the Hausdorff distance between Mi and M 2 , 
where Mi,M 2 € .M [TJ p. 365]. 

Definition 3.1. Given Mi,M 2 G .M, i/ie Hausdorff distance between M\ and 
M 2 is 

H{Mi,M 2 ) = max{ sup d(mi,M 2 ), sup rf(m 2 ,Mi)}, 

miGA/i m 2 el/ 2 

where, for S €1 M., 

d(y,S) = inf ||y-m||. 

Next, we review the concept of recession cones |T3l P- 8] and the fundamental 
definitions of contingent cones (TJ p. 121] and proximal normal cones [181 p. 170]. 

Definition 3.2 (Recession cone). For a nonempty subset S o/R p , the extended 
recession cone S + is defined by: 

S+ = {y G W s.t. 3h k -> 0+, 3y fe G S, h k y k -> y}. 



Definition 3.3 (Contingent cone). Let S be a nonempty subset of some normed 
linear space Z and let z G S*. Then, the contingent cone Ts{z) to S at z is defined by: 

T s (z) = {veZ s.t. 3h k -> 0+, 3v fc -> v, z + ft fe v fe G S}. 



Definition 3.4 (Proximal normal cone). Let S be a nonempty subset of some 
Hilbert space Z with inner product (•,•) and let z € S. Then, the proximal normal 
cone Ns(z) to S at z is defined by: 

N s (z) = {v G Z : 3M > s.t. (v,z- z) < M||z-z|| for all z G S}. 
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Note that we use the notation Ng for the proximal normal cone instead of the tradi- 
tional notation Ng to avoid any possible confusion with the ordering cone P. 

Using contingent cones and proximal normal cones, we can respectively introduce 
the concepts of contingent derivative [H p. 181] and coderivative [2] for set-valued 
maps. 

Definition 3.5 (Contingent derivative of set-valued maps). The contingent 
derivative of F at (x, y) is the set-valued map from X to R p defined by 

Graph(DF(x, y)) = T G 

where T is the contingent cone of Graph(.F) at (x,y), as defined above. 

Note that the contingent derivative DF(x, y) is a closed-valued map. 

Definition 3.6 (Coderivative of set- valued maps) . The coderivative DF*(x., y) 
of F at (x, y) is the set-valued map from TV to X defined by: 

#F*(x,y)(w*) = {v* G X, (v*,-w*) G iV Graph(i r)(x,y)}, 

where -/VQ raph (^) is the proximal normal cone of Giaph(F) at (x, y), as defined above. 

Finally, we will also need the notion of Lipschitz [TJ p. 41] and outersemicontinu- 
ous [H>J p. 152] set-valued maps. 

Definition 3.7 (Lipschitz set- valued maps). The set-valued map F is said to be 
Lipschitz around xdi/ there exists a positive constant I and a neighborhood O of 
x such that 

Vxi,x 2 G O, F(x x ) C F(x 2 ) +Z||x x -x 2 ||B. 

For the definition of outersemicontinuity, we assume that X is finite dimensional. 
For our purposes, we will have X = R n+1 . 

Definition 3.8 (Outer semicontinuous set-valued maps). The set-valued map F 
is said to be outer semicontinuous at x if 

F(x) D limsupF(x') := {y s.t. 3x fc -> x, 3y k -> y with y k G F(x fc )}. 

X'— S-X 

4. Multiobjective Optimization. In this section, we first discuss the concept 
of optimality in multiobjective optimization and introduce the notion of generalized 
Pareto optimal elements or minimal elements for a nonempty subset S of R p . We 
next provide several properties related to these elements. We finally conclude by 
studying the Lipschitzian property of the map E : 2 RP — > 2 RP which associates to 
each nonempty subset S of R p the set of minimal elements of S. 

4.1. Optimality in multiobjective optimization. For an optimization prob- 
lem with a p-dimensional vector- valued objective function, the definition of an optimal 
solution requires the comparison of any two objective vectors yi , y 2 in the objective 
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space, which is the set of all possible values that can be taken by the vector-valued 
objective function. This comparison is provided by a binary relation expressing the 
preferences of the decision maker. In this paper, we consider the binary relation de- 
fined in terms of an ordering cone P C IV, which is defined as a nonempty pointed 
convex cone containing the origin [19J . We will additionally assume that P is closed 
with a nonempty interior, i.e., int(P) ^ 0. 

Definition 4.1. Let yi,y2 G IV. Then, yi ^ y 2 if and only ify 2 G yi + P. 

The binary relation in Definition 14.11 yields the definition of generalized Pareto 
optimality. 

Definition 4.2. Let S be a nonempty subset of IV. An element yi G S is said 
to be a generalized Pareto optimal element or a minimal element of S if and only if 
there is no y 2 G S (y 2 ^ yi) such that yi G y 2 + P, or equivalently, if and only if 
there is no y 2 such that yi G y 2 + P\{0}. The set of minimal elements of S is called 
the minimal element set and is denoted by £(S,P). 

Definition 14.21 can be rewritten as follows: 



It is possible to derive a necessary condition for generalized Pareto optimality in terms 
of contingent cones. 

Lemma 4.3. Let S be a nonempty subset of IV and y G £(S,P). Then. 



Proof. Assume that there exists d G int(P) such that — d G Ts+p(y). Then, there 
exist sequences hk — > + and dfc — > — d such that Vfe, y + /ifcdfc G S + P. For large 
enough k, —dfc G int(P), therefore —hkdk G int(P). Hence, y G S + P + int(P) C 
S + int(P), which contradicts the fact that y G £ (S, P). □ 

Condition ()4.1[) yields the definition of the following stronger optimality notion 



Definition 4.4 (Generalized proper Pareto optimality). Let S be a nonempty 
subset of IV . An element y G S is said to be a generalized properly Pareto optimal 
element or a properly minimal element of S , if and only if y is a minimal element of 
S and the zero element is a minimal element of the contingent cone Ts+p(y), i.e., 



ye£(S,P)^(y-P)nS = {y}. 



T s +p(y) n -int(P) = 0. 



(4.1) 



ma pp. 108-109 



T s+P (y)n-P = {0}. 



(4.2) 



The set of properly minimal elements of S is called the properly minimal element set 
and is denoted V£(S, P) C £ (S 1 , P). 
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4.2. Some properties of the minimal element set. A critical role in this 
paper is played by the external stability or domination property [171 pp. 59-66]. 

Definition 4.5 (External stability). A nonempty subset S of HP is said to be 
externally stable if and only if 

Sc£(S, P) + P. 

An immediate consequence of the external stability property is that S + P = 
£(S,P) + P. A sufficient condition for a nonempty closed set S to be externally 
stable is given in Proposition 14.61 Note that this condition is also sufficient to guar- 
antee the existence of minimal elements. 

PROPOSITION 4.6 (Theorem 3.2.10, [171 P-62]). Let S be a nonempty closed sub- 
set of HP . If S is P-bounded [17[ p. 52], i.e., S + (1 — P — {0}, then S is externally 
stable. 

Corollary 4.7. Let K £ K. Then K is externally stable. 

Proof, if is a compact set, hence K + = {0} (Lemma 3.2.1, jTTJ p. 52]). □ 

Having defined the notion of minimal elements, we can now introduce the concept 
of generalized contingent epiderivative for set- valued maps (5j [11] , which derives from 
the concept of contingent derivative for set-valued maps. 

Definition 4.8 (Generalized contingent epiderivative of set- valued maps). The 
generalized contingent epiderivative of F at (x, y) £ Graph(.F') is the set-valued map 
from X to HP defined by 

Vve X, £» t F((x,y);v)=f( J Di^((x,y);v),P), 

where Ff is the set-valued map from X to R p defined by Vx £ X, Ff(x) — F(x) + P. 
When DFf((x.,y);v) is empty, we set DfF((x.,y);v) = 0. 

It is possible that the set Z)i^((x, y); v) does not have any minimal element. In 
such a case, DfF((x, y); v) is just the empty set. 

We conclude this section by a lemma that will be used in <j5] 

Lemma 4.9 ([7]). Let K 1 ,K 2 £ K satisfying K\ C K 2 and K 2 C K\ + P. Then 
£(K 1 ,P)=£(K 2l P). 

4.3. Lipschitzian properties. In this section, we analyze the map E(-) : 2 RP — ► 
2 RP which associates to each nonempty subset S of R p its minimal element set £(S, P): 

VSe 2 RP , E(S) = £(S,P). 

More precisely, given K\,K 2 £ /C, we show that, under suitable conditions, there 
exists M > such that 



(4.3) 
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First, we record some elementary consequences of our standing assumption that 
P is a pointed closed convex cone with nonempty interior. 

Lemma 4.10. Let I > 7 and define the set P^jxe R p , B(x, Z) C P}. Then 
Pi satisfies the following properties: 

1. Pi is nonempty. 

2. Pi is closed. 

3. Pi is convex. 

Therefore, the origin ofW has a unique nearest point d ; in Pi, and we have: 

4- di + PcPi. 

5. \\di\\>l. 

6. 11*11= /Hdxll. 

Finally, 

7. Vxe di +P, B(x,Z) C P\{0}. 
Proof. 

1. This follows from the fact that P is a cone with a nonempty interior. 

2. Let Xfc be a sequence in Pi converging to x. Let y e R p such that ||y — x| < Z. 
We can write ||y — xj = | (y — x + x&) — Xfc|| < Z. As x^ e P/, it follows that 
y — x + Xfe e P. Taking the limit and knowing that P is closed yields y £ P. 

3. This follows directly from the convexity of the norm || • ||. 

4. Let d e P and y e R p such that ||y- (d ; +d)|| < Z. We can write ||y- (dj + 
d)|| = || (y - d) - di|| < Z. As d ; e P u it follows that y - d e P. As P is 
convex, we get y = y- d|deP + PcP. 

5. Since P is pointed, ^ int(P). Therefore, ||x|| > Z for every x e p; in 
particular, ||d;|| > Z. 

Pointedness also requires strict inequality here. To see why, suppose ||d;|| = I. 
Consider an arbitrary x such that (d;,x) > 0, and define A = (d;, x)/||x|| 2 . 
Then 

|| Ax - d ; || 2 = A 2 ||x|| 2 - 2A(d ; ,x) + Z 2 = Z 2 - (d ; ,x) 2 /||x|| 2 . 

Hence, ||Ax - d;|| < I and Ax e P. As A > 0, it follows that x e P. This 
shows that every x in the open half-space defined by (d;, x) > lies in P. P 
being closed, we get that P contains a closed half-space, which contradicts 
that the fact P is pointed. 

6. It can easily be seen that p = ZPi. Therefore, ||d;|| = inf{||x||, x e p} = 
Zinf{||x||, xePi} = Z||d 1 || =lfi(P). 

7. This follows directly from (4) and (5). 

□ 

As P has a nonempty interior, there exists a closed ordering cone with nonempty 
interior C such that P C int(C) U {0}. We can therefore define JC(C, P) as the set of 
all nonempty compact subsets K of R p such that 



£(K,P)=£(K,C). 



(4.4) 
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The proof of (|4.3p . completed in Proposition 14.151 requires two technical lemmas. 
Lemma r4.12[ which uses Lemma \4. Ill states the boundedness of a set introduced in 
the proof of Proposition ^. 131 Hereinafter, the complement of a set S C R p is denoted 
S c . 

Lemma 4.11. Let a(C,P) = mf{||x-y||, x e P, y G (int(C)) c }, where P = 
{dGP, ||d||=l}. Then a(C, P) > 0. 

Proof. Assume for contradiction that a(C, P) — 0. Then there exist two sequences 
Xfe in P and yu in (int(C)) c such that lim ||xfc — y^l =0. As P is compact, by passing 
to a subsequence if necessary, we can assume that X& converges to some xeP. There- 
fore, the sequence yt is bounded. By passing to a subsequence if necessary, we can 
assume that y^ converges to some y. Clearly, y = x; also, y € (int(C)) c as (int(C)) c is 
closed. By assumption, P C int(C)U{0}. It follows that y G (int(C)U{0})n(int(C)) c , 
which implies y = and therefore x — 0, which contradicts xeF. D 

Lemma 4.12. Let y G (int(C)) c . Then, Vx G (y + P) n (int(C)) c , ||x|| < 
a'(C*,P)||y||, where a'(C,P) = (1 + l/a(C, P)) > 1. 

Proof. Let x E (y + P) n (int(C)) c . Choose deP such that x = y + d. If d = 0, 
then x = y and ||x|| = ||y|| < a'(C, P)\\y\\ as a'(C,P) > 1. Otherwise, we can write 
x/||d|| - d/||d|| = y/||d||. We have d/||d|| e P, x/||d|| e (int(C*)) c , therefore from 
LemmalSIl ||y||/||d|| > a(C,P). Finally, ||x|| < ||y|| + ||d|| < (1 + l/a(C, P))||y||. □ 

PROPOSITION 4.13. There exists a constant M(C,P) > such that MK l7 K 2 € 
/C(C,P), 

H{E(K x ,P),E{K it P)) < M(C,P) H(K U K 2 ). 

Proof. If H(Ki, K 2 ) = 0, then Ki — K 2 and the result is obvious. Therefore, 
assume that H{K U K 2 ) > and let I = U{K ll K 2 ). Let yi G £{K 1 ,P). Then by 
definition of /, there exists k 2 G such that |jyi — k 2 || < /. From Corollary 14.71 
K 2 is externally stable, hence there exists y 2 G £(K 2 ,P) such that k 2 G y 2 + P. 
From Lemma OPT 7). there exists d; 6 C such that Vx G d; + C, P(x,Z) C C\{0}. 
We prove now that yi — y 2 — d( G (int(C)) c . Assume that yi — y 2 G d; + C, then 
B(yi - y 2 ,0 G C\{0}, and therefore B(y 2 ,l) C yi - C\{0}. By definition of Z 
again, there exists ki G K\ such that ki G B(y 2 ,l). Hence, yi G ki + C\{0}. 
But, yi G £{Ki,P) and by (JOJ), B{K U P) = £{Ki,C). Therefore, we obtain a 
contradiction. Finally, yi — y 2 ^ d; + C. Hence, yi — y 2 — d; G (int(C)) c . Now, 
we want to use Lemma 14.121 with y = yi — k 2 — d; and x = yi — y 2 — df. As 
x G (int(C)) c and x — y = k 2 — y 2 G P, we only need to check that y G (int(C)) c . If 
yi — k 2 G d( + C, then by definition of d;, ||yi — k 2 | > I, which is a contradiction. 
Hence, y x — k 2 ^ d; + C, or yi — k 2 — d; G (int(C)) c . From Lemma T4.121 it follows 
that 

||y 1 -ya-d,||<o , (C,P)||yi-k 2 -d I ||. 

Hence, 

||yi-y 2 || < (l + a / (C J P))||d,|| + a / (C,P)||yi-k 2 ||. 
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From Lemma |4. lOf 6) . we have ||d;|| = lfi(C). Recalling that ||yi — k 2 || < I, and 
defining M(C, P) = (1 + a'(C, P))fi(C) + a'(C, P), we get 



||yi-y 2 || <M(C,P) I, 

which shows that 



sup d{ yi ,£(K 2 ,P)) < M(C,P) U{K U K 2 ). 
yi e£(K u P) 

Interchanging the role of K\ and K% finally yields 

U(£(K U P),£(K 2 ,P)) < M{C,P) H(K U K 2 ). 

□ 

We conclude this section by proving in Proposition 14.141 that (|4.4[) implies that 
£(K, P) = V£(K, P). In other words, a set K G IC(C, P) only contains properly min- 
imal elements. 



Proposition 4.14. If K e K{C,P), then £{K,P) =V£{K,P). 

Proof. By definition, we have T>£{K,P) C £(K,P). Let y G £{K,P). By 
definition of JC(C, P), it follows that y G £ (K, C). Assume that y £ V£(K, P). Then 
there exists d € P, d ^ 0, such that -d G T K+P (y). Now, as P C int(C) U {0} 
and d ^ 0, it follows that d e int(C). As K + P C K + C, T K+P (y) C T K+C (y). 
Therefore, d G Tk+c{y) H — int(C), which, from Lemma [4.3[ contradicts the fact 
that y e £(K,C). Q 

5. A multiobjective dynamic programming equation. In this section, we 
state the multiobjective dynamic programming equation satisfied by the set-valued 
return function V. For this purpose, we need to introduce some additional notation. 
Let t € I, t € (0,T — t], x g R™, and define the bounded set 

Y(t, t, x) = I J(t, t + T,x, u(-)) + V(t + r, x(t + r; i, x, u(-))), u(-) G W 

PROPOSITION 5.1. For each t 6 I, r 6 (0,T — t], x G R", i/ie multiobjective 
dynamic programming equation for (MOC) is: 

y(*,x)=£(cl(y(r,t,x)),P), 

or, using the definition ofY{r, i,x) above, 

V(t, *)=£ (d(S.J(t, t + r, x, u(-)) + y(i + r, x(i + t; t, x, u(-))), u(-) G U j \ , P 

(5.1) 

When t = T — t, using the fact that V(T, ■) = {0}, it can be seen that (|5.1I) reduces 
to the definition of V. Therefore, assume that r < T — t. We prove Proposition 15.1 
at the end of this section using the following three lemmas. 

Lemma 5.2. ?(r,t,x) c cl(y(i,x)). 

Proof. Let y G Y(t, £,x) and e > 0. Then there exist u(-) G U and y G Y(t + 
t, x(£ -(- t; t, x, u(-))) such that 

||y-J(* > i + r,x,u(-))-y||<E. (5.2) 
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Moreover, we have y = J(£ + r, T, x(£ + r; £, x, u(-)), u(-)) for some u(-) £ U. Define 
the new control u(-) £lA as 



u( S ) 

Observe that 



u(s), £<s<£ + r, 
u(s), £ + t<s<T. 



and 



L(x(s;£,x, u(-)),u(s)) ds, 

= J(£ + t,T,x(£ + t;£,x,u(-)),u(-)) 

= / L(x(s; £ + r, x(£ + t; £, x, u(-)), u(-)), u(s)) ds 

= / L(x(s; £ + r, x(£ + r; £, x, u(-)), u(-)), u(s)) ds. 

J t + T 



Hence, 



J(t, i + r, x, u(-)) + y = J(t, T, x, 5(0) G Y(t, x). 
Since e > is arbitrary, (15.21) implies that Y(r, £,x) C c\(Y(t,x)). □ 

Lemma 5.3. F(t, x) c Y(t, t, x) + P. 

Proof. Let y £ Y(t,x), We can write y = J(t,t + r, x, u(-)) + y with y G 
Y(t + t, x(t + t;£,x, u(-))). As cl(F(t + r, x(i + r; £, x, u(-)))) is externally stable 
(Corollary 1277)) . there exist y* G V(t + r, x(£ + r;£,x, u(-))) and d £ P such that 
y = y* + d. Therefore, 

y = J(£,£ + r,x,u(-))+y*+d, 
with J(i, £ + t, x, u(-)) + y* G F(t, £, x). Hence, y G Y(t, £, x) + P. □ 

Lemma 5.4. cl(?(r, £, x) + P) c c1(?(t, £, x)) + P. 

Proof. This is a consequence of the facts that the set cl(F(r, £,x)) is bounded 
and P is closed. □ 



We can now proceed with the proof of Proposition 15.11 

Proof. [Proposition 15.1] Apply Lemma [4.91 with K\ = c\(Y(t, £,x)) and = 
cl(F(£, x)). The sets cl(F(r, £, x)) and cl(Y(£, x)) are compact. The inclusion K\ C K2 
comes from Lemma 15.21 while the inclusion K2 C K\ + P comes from Lemmas 15.31 
and KM □ 



Remark 5.1. The dynamic programming eguation (|5.1|) obtained in Proposi- 
tion 15.11 is a generalization of the dynamic programming equation obtained for single 
objective optimal control problems [2lll8j. Indeed, when p = 1 and P = R+, and using 
Remark 12.11 both sets in (15.11) contain exactly one element, so (|5 . 1 [) is equivalent to 

rt+T 

w(£,x) = inf / L(x(s; £, x, u(-)), u(s)) ds + v(t + r, x(£ + r; £, x, u(-)))- 
u(-)euj t 
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6. Generalized contingent solution for (MOC). The notion of lower Dini 
solution to the Hamilton- Jacobi Equation for finite-horizon single objective optimal 
control problems was introduced in [TSJ p. 454] (see also [5J p. 127]). In this section, 
using the concept of contingent derivative for set- valued maps, we extend this notion 
to (MOC). We call this extended notion generalized contingent solution for (MOC). 

6.1. Definition. Our definition of generalized contingent solution for (MOC) 
assumes set-valued maps of a particular type, as described in Definition 16.11 This 
assumption will be used in Corollaries 18 . 411531 to state that the set- valued return func- 
tion V is the unique generalized contingent solution for (MOC). 

Definition 6.1. A set-valued map W from I x R n to R p is said to be an extremal 
element map if, for all (f,x) G / x R™, for all y G W(t, x), 



W(t,x)n(y-P) = {y}, 



3.1) 



W(t,x)n(y + P) = {y}. 



3.2) 



Let (FL)(x) = cl(co({(f (x, u), L(x, u)), u £ U})), where co(S') denotes the con- 
vex hull of the set S. 

Definition 6.2. An extremal element map W from I x R™ to TV is said to be 
a generalized contingent solution for (MOC) if: 

• For all (t,x) G [0,T) x R". for all y £ W(t,x), there exists (f,L) £ (FL)(x) 
such that 

-LeM t ((t,x,y);(l,f)). (6.3) 

• For all (i,x) £ (0,T] x R n , for all y £ W(t,x), and for all (f,L) £ (FL)(x), 

LGDW t ((t,x,y);(-l,-f)). (6.4) 

• For all xeR", 

W(T,x) = {0}. (6.5) 

6.2. A reformulation for (f6T3|) . When W is Lipschitz around (t,x) £ (0,T) x 
R" and y G W(t,x) is a properly minimal element, i.e., y G T'£(W(t, x), P), there 
exists a more compact formulation for 



Proposition 6.3. Let W be a generalized contingent solution for (MOC), 
(t,x) £ [0, T) x R n , and y £ W(t, x). Assume that W is Lipschitz around (t,x) 
and that y £ V£(W(t, x), P). Then we have: 



OGf cl L+fltf((t,x,y);(l,f)), (f, L) £ (FL)(x) , P + P, (6.6) 
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We complete the proof of Proposition 16.31 later in this section. Beforehand, we need 
to show that the generalized contingent epiderivatives of W at (t, x, y) in the direc- 
tion (1, f), i.e., D^W((t, x, y); (l,f)) are nonempty. We derive this result in the same 
general setting as in Sj3] 

Lemma 6.4. Let Si, S2 be nonempty subsets of IV. Then, 

1. {c\{Si))+ = St . 

2. If Si C S 2> then St C S£. 

3. Let I > 0. If Si C S2 + ^B and S2 is P -bounded, then cl(Si) P -bounded. 

Proof. The proof of (1) and (2) can be found in [T3J p. 9]. To prove (3), as- 
sume Si C S 2 + IB and let y G (cl(Si))+ n -P = S+ n -P by (1). By defi- 
nition of the recession cone, there exist sequences hk —> + , y k G Si such that 
hkfk -> y- By assumption, we have y fe = y k + lyk, where y k € S 2 and y k G B. Then, 
h k yk = hkYk ~ hklyk y- Hence, yeS 2 + and y G -P, or y e S 2 + fl -P. As S 2 is 
assumed to be P-bounded, it follows that y = 0. □ 

Proposition 6.5. Let (x, y) G Graph(F). Assume that F is Lipschitz around x 
with Lipschitz constant I. Then, we have: 

1. Vvel, £>F((x,y);v) ^ 0. 

2. The set-valued map Z)P(x, y) is Lipschitz with Lipschitz constant I, i.e., 

Vvi,v 2 G X, DF(x,y)(vi) C DF(x,y)(v 2 ) + lB. 

5. Cf t ((xj);0)cT ff(x) ( y ). 
Proof. The proof of (1) and (2) can be found in [TJ p. 186]. To prove (3), let 
w G -Dff((x, y); 0). By definition of the contingent derivative, there exist sequences 
hk — > + , va, — > 0, and — > w such that 

y + /i fe w fc G F(x + h k v k ) + P. 

Using the Lipschitz property, we get 

y + h k w k G P(x) + Zft fe ||v fe ||B + P. 

Hence, 

y + Mw fc - Z||v fe ||y fe ) G F(x) + P 
for some sequence y& G B. We have — Z||vfe||yk — > w, which shows that w G 

TWy). □ 

The conclusion of Proposition ^. 5f l) might fail if F is not Lipschitz, but simply contin- 
uous. Take for example the set-valued map F defined from R to R by F(x) = {x 1 ^ 3 } 
and P = R+. Then, it is easy to check that DF((0, 0); v) = when v ^ 0. 

Proposition 6.6. Let (x, y) G Graph(F) . Assume that F is Lipschitz around x 
and that y G P£(P(x), P). TTien, 

PP t ((x,y);0)n-P = {0}. (6.7) 
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Moreover, Vv G X, the sets PP|-((x, y); v) and Df ((x,y);v) are P -bounded. 

Proof. As -F is Lipschitz, Pf is also Lipschitz. Therefore, from Proposition 
KTSril. PP t ((x,y);0) ^ 0. Now, let w G DP t ((x, y); 0) n -P. From Proposition 
MM3), DF t ((x,y);0) C 2> t(x) (y). Hence, w G Tp t(x) (y) n -P By assumption, 
y G V£(F(x),P). Therefore, w = 0. 

To conclude that the sets DFf((x, y);v) are P-bounded, observe first that the set 
DF-f((x, y); 0) is P-bounded. Indeed, DFf((x, y); 0) is a closed cone, hence DFf((x, y); 0) + = 
DF t ((x,y);0), or £>F t ((x,y);0)+ n -P = DF t ((x,y); 0) n -P = {0}. Now, from 
Proposition 16. 5f 2). the set-valued map PPf(x, y) is Lipschitz. The conclusion there- 
fore follows from Lemma F6.4I T3) . P-boundedness of the sets Df((x,y);v) is readily 
obtained from the inclusion DF((x, y); v) + P C Di^((x,y);v). □ 

The assumption that y is a properly minimal element is essential in obtaining 
that the sets DF^((x, y); v) are P-bounded. If y is only assumed to be a mini- 
mal element, then, using Lemma 14.31 and Proposition 16. 5[ instead of (|6.7j) . we would 
get ZXF-f((x, y); 0) n — int(P) = 0. Therefore, when y is only a minimal element, 
DFf((x, y); 0) is not necessarily P-bounded, as illustrated by the following example. 
Let X = R, p = 2, and P = R^_. Let S = {( yi , yf ) if y x < 0}U {(y x , -y x ) if y x > 0}. 
Take (0,0) G 5. We have that (0,0) is a minimal element. Moreover, T5 + p(0,0) = 
{(wi,w 2 ), w 2 > 0} U {(wi,w 2 ), w 2 > -wi} Hence, T s+P (0, 0) H -int(P) = 0, 
Ps+p(0,0) n -P = {(wi,0), wi < 0} ^ {(0,0)}, and therefore (0,0) is not a prop- 
erly minimal element. Now, define the constant set-valued map: Vx G R, F(x) = S. 
The set- valued map F is obviously Lipschitz around all x G R, and it is not hard to 
show that Va; G R, DF t ((x, 0, 0); 0) = T s+P (0,0). Hence, the set DF t ((x, 0, 0); 0) is 
not P-bounded. 



Proposition 16.61 implies that the sets D^W((t,x, y); (l,f)) appearing in (|6.6[) are 
nonempty. To see this, note from Proposition |L6] that it suffices to show that the sets 
DW^((t,x,y); (l,f)) arc P-bounded. Applying Proposition 16.61 to W achieves this. 



The next step is to show P-boundedness of the set 

clQL + P t W((i,x,y);(l,f)), (f,L) G (FL)(x) 
Again, we derive this result in a general setting. 



Proposition 6.7. Let (x, y) G Graph(F). Assume that F is Lipschitz around x 
and that y G P£(P(x),P). If S is a nonempty compact subset of X x R p , then the 
following set is P-bounded: 

cl( (J w + P t P((x,y);v)V 



Proof. As DfF((x, y);v) C DP|-((x, y); v), from Lemma EH T)-(2), it is enough 
to show that (J^ v w ) e g w + DFf((x, y); v) is P-bounded. Let 

we( |J w + DF t ((xj);v)j n-P. 

Mv,w)6S ' 
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By definition of the recession cone, there exist sequences hk — > + , wj. G (J, v w ) G g w+ 
PPj-((x, y); v) such that /ifcW/. — » w. We have Wj. £ wj. + PPj-((x, y); v^) for some 
(vfcjW/j) e 5. As 5* is compact, by passing to a subsequence if necessary, we can as- 
sume that (vfc,Wfc) — > (v,w) £ S. From Proposition l6.51 DFf(x., y) isLipschitz; let its 
constant be I. Hence, we have £ Wfc+Wfc+Z||vfc — v||B where wj, £ DF^((x, y); v). 
It follows that /ifeWfc — ► w. Hence, w £ DF|((x,y);v) + n —P. But, from Proposi- 
tion 16.61 DFf((x, y); v) is P-bounded, hence w = 0. □ 

To be able to apply Proposition 16.71 in the setting of Proposition 16.31 it suffices 
to show that the set (FL)(x) is a nonempty compact subset of R" x R p . 

Lemma 6.8. The set (FL)(x) is a nonempty compact subset of R" x R p for all 
x G R". 

Proof. The set (FL) (x) is closed by definition. The boundedness follows from the 
boundedness of f, see ()2.2[) . and the boundedness of L, see ()2.5[) . □ 



We can now proceed with the proof of Proposition [6731 

Proof. [Proposition I6.3| Assume that (|6.3|) holds for some (i, x) G [0, T) x 
R™, y G Wit, x), and (f, L) G (FL)(x). As shown in Proposition 16.61 the set 
DW^((f,x,y);(l,f)) is P-bounded, hence externally stable from Proposition 
Therefore, (16.31) implies 



-LGC r W((i,x,y);(l,f)) + P. 

Hence, 



-Pn|L + I> t W((t ) x,y);(l,f)) ) (f,L)G (FL)(x)| ^ 0, 



PnclNL + D t W((t,x,y);(l,f)), (f,L) G (FL)(x) > ) ^ 



Oecl(jL + D t W((t,x,y);(l,f)), (f , L) G (FL)(x)|J + P. 
From Proposition 16.71 together with Lemma 16.81 the following set is P-bounded: 

d^|L + D t W((t,x,y);(l,f)), (f,L) G (FL)(x) 
Hence, it is externally stable from Proposition 14.61 Therefore, 

0G£(cl( U + D t W((t,x,y);(l,f)), (f,L) G (FL)(x) }) ,P ) + P. 
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It is possible to derive a converse to Proposition ^. 31 Again, we derive this result 
in a general setting. 



Proposition 6.9. Under the assumptions of Provosition [6771 assume that 

0€f(d( |J w + O t F((x,y);v)|)|,F||P. 
^ ^ (v,w)es ' ' ' ' 

Then there exists (v, w) G S such that 

-w G DF t ((x,y);v). 

Proof. By hypothesis, some deFobeys-de£ cl (J w+D t F((x,y); v) 

^ Mv,w)6S ' > 

Then -d = lim fc ^+oo w fe + w fe , where w fe G P t P((x, y); v fe ) C PP t ((x, y); v fc ) and 
(vfc,WA;) 6S. As S 1 is compact, by passing to a subsequence if necessary, we can as- 
sume that (vfe, Wfc) — > (v, w) G S. Hence, Wfe converges to — d w. From Proposition 
I6.5f 2), P/P^(x, y) is Lipschitz. Moreover, DFf(x, y) is a closed-valued map, hence it 
has closed graph. Therefore, — d — we PPj-((x, y); v), or — w G PPj-((x, y); v) + d, 
But, DF t ((x,y);v) =Df t ((xj):v)+P. Hence, -w G DF t ((x, y); v). □ 

Corollary 6.10. Let W be an extremal element map, (i,x) G [0,T) x R", and 
yG VF(i,x). Assume that W is Lipschitz around (t,x), that y G V£(W(t,x), P) , and 
that ^Ml holds. Then ([O]) holds for some (f,L) G (FL)(x). 



6.3. A reformulation for (I6.4p . As above for (|6.3p . we show that, when 14 7 is 
Lipschitz around (i,x) G (0,T] x R™ and y G VF(i,x) is a properly minimal element 
with respect to -P and P, i.e., y G V£ (W(t,x), -P) n P£ (W(t, x), P), there exists 
a more compact formulation for ()6.4|) . 



Proposition 6.11. Let W be a generalized contingent solution for (MOC), 
(i,x) G (0,T] x R™, and y G W(t,x). Assume that: 

1. W is Lipschitz around (i,x), 

2. y G P£ (W(t, x), -P) n V£(W(t, x), P), 

5. i/ie set-valued map D^W(t,x., y) is outer semicontinuous, 
4- the set £ (cl(S'), — P) is closed, where 



S=^ -L + D t W((t,x,y);(-l,-f)), (f , L) G (FL)(x) 

TTien we have: 

Oeffclfj -L + DtW((t,x,y);(-l,-f)), (f, L) G (FL)(x) U , -P ) + P, (6.9) 



Knowing that y G V£(W(t, x), P) and using Proposition 16.61 it is possible to 
conclude that the sets .DW-|-((t, x, y); (—1, — f)) are P-bounded. Hence, (|6.4p becomes 



LGP t ^((t,x,y);(-l,-f))+P 
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or 

V(f, L) G (FL)(x), G — L + D t W((t, x, y); (-1, -f )) + P. (6.10) 

Hence, proving Proposition 16.111 amounts to proving that (|6.10l) implies (|6.9[) . We 
provide a constructive proof below, but beforehand, we need the following two re- 
sults, where Proposition 16. 12l is derived in our usual general setting. 

PROPOSITION 6.12 (Theorem 2.1, [17J. Let (x, y) G Graph(F). Assume that P 
has a compact base. Then, 

Vv G X, D t P((x,y);v) C DP((x,y);v). 

Lemma 6.13. Under the assumptions of Proposition [6.111 the set cl(5) ; where 5 
is defined by (|6.8[) is —P '-bounded. Moreover, if w G £(cl(5), — P), i/ien i/iere exists 
(f,L) G (FL)(x) suc/i that we -L + D t W((t,x,y); (-1, -f)). 

Proof. From Lemma T6. 41 (l)-(2) and Proposition 16.121 it is enough to show that 
the following set is — P-bounded: 

|-L + £»W((t,x,y);(-l,-f)), (f,L) G (FL)(x)J. 

From the last statement of Proposition 16.61 applied with — P, we get that the sets 
W((t,x,y);(-l,-f)) arc —P-bounded. From there, the proof follows the same 
lines as the proof of Proposition 16.71 

Let w t £ Sbea sequence converging to w. We have w k G — L k +D^W((t 7 x, y); (— 1, — f fe 
for some (ffe,Lfe) G (FL)(x). As (FL)(x) is compact, we can assume, by passing to a 
subsequence if necessary, that (ffe,Lfe) converges to (f , L) G (FL)(x). By assumption, 
DfW(t,x,y) is outer semicontinuous. Hence w + L G DfW((t, x, y); (— 1, — f)), or 
w6-L + fl t f((i,x,y);(-l,-f)).D 

We can now proceed with the proof of Proposition 16.111 Recall that 5 is defined 
by (EU). 

Proof. [Proposition ^. llj We build sequences ffe, d^, and w^, as follows. Take 
(ffc,L k ) G (FL)(x). From dUni), there exists d fc G (-L fc +D t W((t,x,y);(-l,-f fc )))n 
—P. Hence, dfe G S. If d^ G £ (cl(S), — P), then the proof is complete. Other- 
wise, there exists G cl(5) such that G dfe + P\{0}. As 0-1(5) is externally 
stable with respect to — P, without loss of generality, we can assume that Wfe G 
£(cl(5),— P). From Lemma T6.131 there exists (ffc+i, Lfc+i) G (FL)(x) such that 
wfe G -Lfc+i + D^W((t,x,y);(-l,-ffc + i)). If w k G -P, the proof is complete. 
Otherwise, Wfe G — P c . Repeat now the procedure by choosing dfe + i G (— Lfc +i + 
DfW((t,x, y); (— 1, — ffc+i))) fl — P using (|6.10p . To summarize, the sequences ffe, Lfc, 
dfe, and Wfe, satisfy: 

Start with (fo,L ) G (FL)(x), d G (-L + D t W((t,x,y); (-l,-f ))) fl -P. Then, 



1. Wfe G d fe + P\{0}, 

2. Wfe Gf(cl(5),-P), 
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3. w fe G -L fe+1 + D t W{{t,x,y);(-l,-f k+1 )) for some (f fe+1 ,L fc+1 ) G (FL)(x), 

4. w fe G -P c , 

5. d fc+1 G -L fc+ i+£> t W((t,x,y);(-l,-f fc+ i)), 

6. d fc+1 G -P. 

As (FL)(x) is compact, by passing to a subsequence if necessary, we can assume that 
(ffc,Lfc) converges to (f, L) G (FL)(x). Consider now the recession cone of the set 
{(dfe.Wfc), k G N}. Take (d,w) G {(d fe ,w fc ), k G N}+, i.e., let h k -> 0+ be such 
that /ifcdk — > d and /ifcWfc — >• w. From (1) and (6) above, we get 

d-we-P, (6.11) 



d G -P, (6.12) 

We show now that necessarily (w, z) = (0,0). From Proposition 16.121 we have 

D t W((t, x, y); (-1, -f fc+1 )) C DW((t, x, y); (-1, -f k+1 )). Hence, w fe +L fc+1 G DW((t, x, y); (-1, -f fc+ i)). 

Moreover, from Proposition I6.5[ the set- valued map DW(x., y) is Lipschitz. Hence, 

along the lines of the proof of Proposition l6?7l it can be shown that w G DW((t, x, y); (— 1, — f )) + . 

A similar argument also gives d G DW((t, x, y); (— 1, — f)) + . Hence, from (|6.12[) . 

d G DW((t, x, y); (-l,_-f ))+ n -P, but we know that the set DW((t, x, y); (-1, -f)) 

is P-bounded, hence d — 0. Therefore, from (|6.11[) . we get that w G P. Hence, 

w G DW((t, x, y); (-1, -f))+ n P. But, we know that the set DW((t, x, y); (-1, -f )) 

is —P-bounded, hence w = 0. Finally, (d,w) = (0,0), and therefore the set 

{(dfc,Wfc), k G N} is bounded (Lemma 3.2.1 [16l p. 52]). 



By passing to a subsequence if necessary, we can therefore assume that d^ converges to 
d and that converges to w. By assumption, DfW(t, x, y) is outer semicontinuous, 
hence d + L, w + L G D^W((t, x, y); (—1, — f)). From (1) above, we get d — w G — P, 
or d + L — (w + L) G —P. d + L and w + L are both minimal elements, hence we 
must have d + L = w + L, or d = w. From (6) above, we get d G — P, hence w G —P. 
Finally, recall that w is the limit of the sequence Wk G £(c\(S), — P), which is a closed 
set by assumption. Hence, w G £(cl(S), — P) n — P, which completes the proof. □ 

6.4. The case p = 1. In this section, we show that, when p = 1 and P = R + , 
the notion of lower Dini solution for single objective optimal control problems is re- 
trieved from Definition 16.21 In this case, W is a set-valued map from / x R" to R. 
As, by definition, W is an extremal element map, it follows from (|6.1j) and (|6.2j) that 
the values of W reduce to singletons. Hence, we can write W(t, x) = {w(t, x)}, where 
w is a function from / x R" to R. 



To go further, we need the following proposition which relates the contingent epi- 
derivative and the lower generalized Dini derivative for extended real- valued functions. 

Proposition 6.14 ([IJ p. 225]). Let (t,x) eJxR" and (r,v) G R ,l+1 . Then, 
by definition of the contingent epiderivative Dfw(t,~x) : R™ +1 — > R U {— oo, +oo}: 

Vu G Dw t ((t, x); (r, v)), D t w((t, x); (r, v)) < u. (6.13) 

Moreover, 

D^w{{t,x); (t,v)) = limmf . 

h-*0+,(s,p)-*(T,v) h 
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Hence, 

D t w((t, x); (r, v)) = d~w((t, x); (r, v)), (6.14) 

where d~w((t, x); (r, v)) denotes the lower generalized Dini derivative of w at (t, x) 
in the direction (t, v) . 

As LW t ((t,x,w(i,x)); (r,v)) = Dto t ((i,x); (r, v)), therefore writes 

-l6%((t,x);(l,f)). 

Hence, from (|6TT5]) - (|6TT3]l . 

fl-«;((t,x);(l ) f)) < — L, 

or 

inf L + 9-io((t,x);(l,f)) < 0. (6.15) 

(f,L)e(FL)(x) 

Accordingly, (|6.4I) writes 

V(f , L) G (FL)(x), L G %((t, x); (-1, -f)). 
Hence, from (j6T3|l - (j6T4|l . 

V(f , L) G (FL)(x), 5" W ((t, x); (-1, -f )) < L, 



sup -L + S"«)((t,x); (-l.-f)) < 0. (6.16) 

(f,L)e(FL)(x) 

Finally, 

w(T,x)=0. (6.17) 

Equations (|6.15|) - (|6.1 T[) precisely correspond to the notion of lower Dini solution for 
single objective optimal control problems. 



Equations (|6.15p and (|6.16|) can also be directly obtained from Propositions 
and l6.ll! First, observe that w(t,x) is a properly minimal element of the set W(t,x) 
with respect to -P and P, i.e., tu(t,x) G VE (W(t, x), -P) n P£ (W(t, x), P). Assume 
now that is Lipschitz around (<,x), then it follows that id is also Lipschitz. Hence, 
from Propositions 16.61 and 14.61 the sets DfW((t, x, w(t,x)); (r, v)) are nonempty and 
reduce to singletons. More precisely, we have 

D^W((t, x, w(t, x)); (r, v)) = {D t w{{t, x); (r, v))}. 

Hence, (|6.6[) from Proposition 16.31 yields 



0££ cl LlM((,x);(l,f)), (f,L)G (FL)(x) ,P + P 



20 



A. GUIGUE 



which is precisely (|6.15[) . For Proposition 16.111 note that Assumptions 3 and 4 are 
automatically satisfied. Indeed, the set-valued map D^W(t, x, w(t, x)) is Lipschitz 
and takes closed values, i.e., singletons, hence it is outer semicontinuous. Moreover, 
the set £(c\(S),—P) in Assumption 4 reduces to a singleton, hence it is closed. Finally, 
(El yields 



0g£I ell |-L + 0-tfl((t,x);(-l,-f)), (f,L) G (FL)(xH 1,-Pl + P, 

which is precisely (|6.16[) . 

7. The set-valued return function V is a generalized contingent solu- 
tion for (MOC). In this section, we first show that the set-valued return function is 
a generalized contingent solution for (MOC). We then prove that the set-valued map 
is outer semicontinuous. 

Proposition 7.1. The set-valued return function V is a generalized contingent 
solution for (MOC). 

Proof. First, we need to show that V is an extremal element map. The fact that 
(|6.1|) and (|6.2j) hold follows directly from the definition of V. It remains to show that 
V satisfies (|6.3p - (|6.5[) . The proofs for (|6.3p and (|6.4I) are respectively postponed to 
£17.11 and £17.21 For (|6.5[) . this again follows directly from the definition of V. □ 

Proposition 7.2. The set-valued map Vf is outer semicontinuous on I x R". 
Proof. Let (t,x) G / x R™. From Corollary 14. 7\ the set cl(y(i,x)) is externally 
stable. Hence, 

V(t,x) + P = c\(Y(t,x)) + P. 

Therefore, showing that the set-valued map is outer semicontinuous at (t, x) 
amounts to showing that the set-valued map cl(Y (■,■)) + P is outer semicontinu- 
ous at (t, x). But, this follows from the fact that the set cl(Y(t, x)) + P is closed and 
the set- valued map Y is Lipschitz fCorollary |2.3l) . □ 



7.1. V satisfies (l(i.3[) . For the developments below, we need the two estimates 
contained in the following lemma. 

Lemma 7.3. For (t, x) G [0, T) x R™ and u(-) G U, one has 

- f + f (x(s; t, x, u(-)), u(s)) ds = - / + f (x, u(a)) ds + o(l), (7.1) 



and 



^- J L(x(s;t,x,u(-)),u(s)) da=^y L(x, u(s)) ds + o(l), (7.2) 

as t -> + independently ofu(-). 

Proof. The first estimate (|7.ip follows from the Lipschitz assumption, see (|2.ip . 
and the boundedness assumption, see ()2.2p . on f, while the second estimate (|7.2p fol- 
lows from the Lipschitz assumption on L, see (|2.6p . and the boundedness assumption 
on f, see (gUJ. □ 
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Let (i,x) G [0,T) x R" and y G V(t,x). We have y G V(i,x) C c\(Y(t,x)). 
Hence, Ve^ — > + , there exists y^ G Y"(i, x) such that ||y — y k \\ < e k . Choose a 
sequence T k — > + such that e^/rfc — >• + (e.g. = y^ife). From Lemma 1531 there 
exists Ufc(-) 6M such that 



L(x(s; t, x, Ufc(-)), u fc (s)) ds + V(t + r fe , x(i + r fc ; i, x, u fc (-))) + P, 
or 

yey L(x{s;t,K,u k (-)),u k {s))ds+V{t+T k ,x(t+T k ;t,x,u k (-)))+P+e k B. (7.3) 



From (|7TT|) and (f772|) . we have: 



f(x, Ufc(s)) ds + r fe o(l), 



and 



L(x(s;t,x, u fc (-)),u fe (s)) ds = J L(x, u k (s)) ds + 7*0(1), 



independently from !!&(•). Moreover, 

- / L(x, u fe (s)) ds, - / f (x, u fe (s)) ds G (FL)(x). 

From Lemma T6.81 the set (FL)(x) is compact. Hence, by passing to a subsequence if 
necessary, we can assume that there exists (f , L) G (FL)(x) such that 



^ rt+r k ^ rt+r k \ _ 

— / L(x, Ufc(s)) ds, — / f (x, Ufc(s)) ds I — > (f . L) as k — >• oo. 



ft + T k -| ft+Tk 

s T k 

Hence, we can write 

X{t + Tk',t, X, Ufc(-)) = x + T k ({ + o(l)), 

and 

L(x(s;t,x, u fc (-)),u fc (s)) ds = r fe (L + o(l)). 

Substituting these two equalities into (|7.3[) yields 

y + r fe (-L + o(l)) G y(f + r fe ,x + r fc (f + o(l)))+P + e fe B, 

or using the fact that e k = o(rfc), 

y + r fe (-L + o(l)) G y(t + r fe ,x + r fe (f + o(l)))+P. 

But this precisely corresponds to the definition of the contingent derivative of Vf at 
(t,x, y) in the direction (l,f). Hence, 

-LG W t ((t,x,y);(l,f)), 

which is (|6.3[) . 
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7.2. V satisfies (|6.4|) . For the developments below, we need the following tech- 
nical lemma. 

Lemma 7.4 ([2, p. 129]). Let (t,x) G (0,T] x R", we have 

(FL)(x) = {g=(f,L)GR"+*: 
g = lim — / (f(x, u fe (s)),L(x, u fe (s))) ds for some r fc -> + and u fc (-) eu\. 

k^oo Tfc Jf_ Tfc J 

Let (i,x) G (0,T] x R n , u(-) G and y G F(t,x). Let 5 -> z(<5; 0, x, u(-)) be the 
solution to 

f z(<5) = -f(z(<5),u(t-5)), 0<<5<r, 
\ z(0) = x. 

The function S — > z(S; 0, x, u(-)) and the trajectory s — >• x(s; t — r, z(r; 0, x, u(-)), u(-)) 
satisfy 

x(f -S;t-T, z(r; 0, x, u(-)), u(-)) = z(8; 0, x, u(-)), < S < r, 

and therefore, x(t; t — r, z(r; 0, x, u(-)), u(-)) = z(0; 0, x, u(-)) = x. Hence, applying 
Lemma [5T2l to the trajectory s — > x(s; t—r, z(r; 0, x, u(-)), u(-)) on the interval [t — r, t] 
gives 

f L(x(s;t-r,z(r;0,x,u(-)),u(-)),u(s)) ds + ^(t,x) C d(y(i-r,z(r;0 ) x,u(-)))) ) 
or, as cl(F(i,x)) is externally stable from Corollary 14.71 



f L(x( S ;i-r,z(r;0,x,u(.)),u(.)),u( S )) ds + V(t,x) C V(t- r, z(r; 0, x, u(-))) + P. 

(7.4) 

Now, let (f, L) e (FL)(x). From Lemma l74l there exist Tfc — > + and Ufc(-) such 
that 

1 /"* .If* 
f = lim — / f(x, u^,(s)) ds, and L= lim — / L(x, Ufc(s)) ds. 

fc^oo Tfc Jt— r fc fe^oo Tfc J t _ Tfc 

We have: 

z(r fe ;0,x, u fe (-)) = x- / f(z(<5; 0, x, u fe (-)), u fe (t - 6)) dS. 
Jo 

As in £17.11 it can be shown that 

z(t;0,x,u(-)) =x-7*(f + o(l)), 

and 

t 

L(x(s;t - r fe ,z(r fe ;0,x, u fc (-)),u fc (-)),u fc (s)) ds = r fc (L + o(l)), 

t-Tfj 
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as r k — ► + independently of Ufc(-). Substituting these two equalities into (|7.4[) yields 
Tfe(L + o(l)) + V{t, x) e V(t - n , x - Tfc(f + o(l))) + P, 

or 

y + r fc (L + o(l)) e V(t-T k ,x-T k (f + o(l))) + P. 

But this precisely corresponds to the definition of the contingent derivative of Vf at 
(t, x, y) in the direction (— 1,— f). Hence, 

V(f , L) e (FL)(x), L e DVf((t, x, y); (-1, -f)), 

which is 

8. Generalized proximal solution for (MOC). The notion of proximal so- 
lution to the Hamilton- Jacobi Equation for finite-horizon single objective optimal 
control problems was introduced in [18} p. 454]. In this section, using the concept 
of coderivative for set-valued maps, we extend this notion to (MOC). We call this 
extended notion generalized proximal solution for (MOC). 

8.1. Definition. Definition 8.1. An extremal element map from I x R n to 
R p is said to be a generalized proximal solution for (MOC) if: 

• For all (i,x) E (0,T) x R™, for all y G W(t,x), and for all w* E W such 
that D*W t (i,x,y)(w*) ^ 0, 

V(f ) v«)eD«ff t (t ) xj)(w'),r + ( inf (v*,f) + (w*,L) = 0. (8.1) 

(f,L)G(FL)(x) 

• For all x £ R" , 

W t (0,x)c limsup Wf{t,x') and W t (T,x) C limsup Wf(i,x'). (8.2) 

t->0+, x'->x t->T~, x'-Kx 

• For a« x e R™, 

W(T,x)={0}. (8.3) 

8.2. The case p = 1. In QQAl we have shown that W(t, x) = {w(t, x)}, where w 
is a function from / x R" to R. Hence, recalling that P = R + , we have gph(H^) = 
epi(w). Therefore, N gph ( Wr) (t, x, w(t, x)) = N cp ^ w )(t, x, «;(*, x)), or 

3iu(t, x) = D*Wt(t, x, w(t, x))(l), (8.4) 

where dw(t,x) is the proximal subdifferential of u> at (i, x) [TH1 P. 135]. Note that, 
again, we use the notation d for the proximal subdifferential instead of the traditional 
notation d p to avoid any possible confusion with the ordering cone P. Substituting 
in (|EU) yields 

V(£*,v*) edw(t,x), e+ inf (v*,f)+L = 0, 

(f,L)e(FL)(x) X 

To retrieve the notion of proximal solution for finite-horizon single objective optimal 
control problems, it remains to show that (|8.2[) yields: 

u>(0,x)> liminf w(t,x') and w(T,x) > liminf io(t,x). 

t->0+, x'->x t-*T~ , x'-+x 

We only prove below the first inequality. From (|8.2|) . there exist t k — > + , Xfe — > 
x, y k -> to(0,x) such that y fc G Wt(*k,xjfe) = {w(t fe , x fc )}+R+. Hence, y fc > w(t k ,x k ), 
from which follows that iz;(0,x) > liminf t _,.Q+ x '_5. x w(t, x'). 
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8.3. Prom generalized contingent solution to generalized proximal so- 
lution. In this section, we show that if a set-valued map is a generalized contingent 
solution for (MOC), then it is also a generalized proximal solution for (MOC). 

Proposition 8.2. Let W be a generalized contingent solution for (MOC). Then 
W is generalized proximal solution for (MOC). 

Proof We first check that flHUJ) is obtained. Let (£*,v*) G D*Wt(f,x,y)(w*). 
Then, (£*,v*,-w*) € N gph{Wr) (t, x, y). Hence, for all (f,v,w) € r gph(Wt) (t,x,y), 

+ (v, v*) + (w, -w*) < 0. (8.5) 

From ([O]) , there exists (f,L) G (FL)(x) such that 

-LeM t ((t,x )y );(l,f)), 

or (l,f, — L) G J 1 gph ( H A t )(t,x, y). Hence, from (|8.5|) . we get: 

r + (f,v*) + (-L,-w*) <0, 



T+ inf (v*,f) + (w*,L) <0. (8.6) 

(f,L)G(FL)(x) X 

Also, from (JB3), for all (f , L) G (FL)(x), 

Lei?W r ((t,x,y);(-l,-f)), 
or (-1, -f,L) G r gph(W / T )(i,x, y). Hence, from (1531) . we get: 

-r + (-f,v*) + (L,-w*) <0, 

or 

T+ inf (v*,f) + (w*,L) >0. (8.7) 

(f,L)e(FL)(x) X 

Combining (gU) and Qj£7$ yields (I8~T|) . 

We turn out to checking that (|8.2j) is obtained. We only check the first condition 
in (|8.2j) . as checking the second condition is similar. Let y G W^(0,x). We have 
y = y + d with y G VF(0,x) and d G P. From (|Ojl . there exists (f,L) G (FL)(x) 
such that 

-LGW t ((0,x,y);(l,f)). 

By definition of the contingent derivative, there exist sequences tk — > 1, ffe — > f, 
Lfe — > L, and /ife — > + such that 

y + hk(-Lk) G VFt(/i/cife,x + /ifcffc). 

Hence, 

y + /ife(-Lfe) G Wf(hktk,X-+hkfk), 

with y + hk{— Lfe) — > y, /ifcife — > + , and x + /ifcffe — > x. This shows that y G 
limsup t _j. + x'^x W^(t,x') and therefore the first condition in (I8.2[) is obtained. □ 
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8.4. Uniqueness. In this section, we first provide in Theorem 18.31 a character- 
ization of set- valued maps satisfying (|8.1[) - (|8.3[) . The proof of this theorem follows 
very closely the proof of Theorem 12.3.7 [T8, P- 460] for single objective optimal con- 
trol problems. Using Theorem 18.31 and the concept of extremal element maps, we 
are able to prove in Corollary 18.51 that the set- valued return function V is the unique 
generalized contingent and proximal solution for (MOC). 

Theorem 8.3. Let W be a set-valued map from I x R" to R p . Assume the 
following for W : 

1. W-f is an outer semicontinuous set-valued map. 

2. Forall(t,x) £ (0,T)xR" ; y g W(t,x.), andw* e W such that D*W t (t,x.,y)(w*) ^ 
0, we have: 

V(r,v*)eD*W t (*,x,y)(w*) ) r+ min (v*,f) + (w*,L) = 0. (8.8) 

(f,L)6(FL)(x) 

3. For all x e R". 

W t (0,x) C limsup W t (i,x') and W t (T,x) C limsup W t (t,x'). 

t->0+, x'^x t->T-, x'-s-x 

4. For all x e R", W(T, x) = 0. 
Then, V(i,x) eJxR", 

K(i,x)cff t (t,x), (8.9) 

and 

W(t,x) C Vf(i,x). (8.10) 

Proof. 

Proof of (|8.9j) : the proof of (|8.9|) uses the Strong Invariance Theorem for Time- 
Varying Systems (Theorem 12.2.4 [HJ pp. 449-452]). 

Let (t, x) £ [0, T) x R" and u(-) € For s e [0, T — t], define: 

• x(s;i,x,u(-)) =x(T-s;t,x,u(-)). 

• a(s;i,x,u(-)) = J T _ s L(x(r;i,x,u(-)),u(r)) dr. 
. (KL)(x)=cl(co({(-f(x,u),L(x,u)), ue(7})). 

• W{s,x) = W(T-s,x). 

It is easy to check that s — > (x(s; t, x, u(-)), a(s; t, x, u(-))) is a solution of the following 
differential inclusion: 

f (x(s),a(s)) e (FL)(x(s)), a.e. ae[0,T-t], 
\ (x(0),a(0)) = (x(T;t,x,u(-)),0). 

Moreover, as (£*>v*) e D* W t (i, x, y)(w*) implies that (-£*,v*) € £)*W ? t (t,x,y)(w*) ) 
from (|8.8|) . we get: 

V(f,L) G (FL)(x), T + (v*,f) - (w*,L) < 0. (8.11) 
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Define now the set-valued map Q from [0, T — t] to R ,l+P by 

Q(s)={(x,a), ae Wr(a,x)}, 

and consider the following constrained optimal control problem: 

(x(s), a(s)) g (FL)(x(s)), a.e. s G [0, T - t], 
(x(0),a(0)) = (x(T;t,x,u(-)),0), 
(x(s),a(s)) G Q(s) for all s G [0,T-i]. 

Our objective is to apply the Strong Invariance Theorem for Time- Varying Systems 
to this problem. To do so, we need to check that the conditions required to apply this 
theorem are satisfied. First, as W(T, x) = by assumption 4, we have Q(0) = R™ x P 
and hence (x(0),a(0)) G Q(0). The condition (x(0),a(0)) G limsup s ^ 0+ Q(s) directly 
follows from assumption 3. Moreover, from the assumptions in the set- valued map 
FL possesses the required properties, and gph(Q) is closed as gph(Q) = gph(W^) 
and Wf is outer semicontinuous by assumption 1. It remains to show the "inward- 
pointing" condition. Hence, let (s.x, a) G gph(Q) with s G (0, T — t) at which 
^gph(Q) ( s ' x ' a ) * s nonem Pty- Take any vector (£* , v* , w* ) G ^gpMQ) ( s j x 7 a ) ? we need 
to show that 

T+ max ((v*,w*),(f,L)) < 0, 

(f,L)e(FL)(x) 

or 

V(f,L) G (FL)(x), T + (v*,f) + (w*,L) < 0. 

As already mentioned, gph(Q) = gph(PF t ). Hence, N gpH ^ (s, x, a) = N gph{W ^(s, x, a), 

and by definition of the coderivative (£*,v*) E D*W^(s, x, a)(— w*). Hence, from 
flHUIJ) , we get: 

V(f,L) G (FL)(x), T + (v*,f) + (w*,L) < 0, 
which is what was needed to show. 

The Strong Invariance Theorem for Time- Varying Systems therefore gives that for 
all (t,x) G [0,T) x R", u(-) G 14, and s G [0,T-t], 

(x(s;t,x,u(-)),a(s;t,x,u(-))) G Q(s), 

or a(s; t, x, u(-)) G Wf(s, x(s; t, x, u(-))). Hence, for s = T — t, 

a(T - t; t, x, u(0) = J(t, T, x, u(-)) G %{T - t, x(T - t; t, x, u(-))) = W t (t, x). 

This holds for all control u(-) G U. Hence, the objective space Y(t,x) satisfies the 
inclusion 

y(*,x) c w t (t,x), 

or, as W-j- takes as values closed sets, 

V(t,x) C cl(V(t,x)) C W t (i,x). 
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From assumption 4 and recalling that V(T, x) = {0}, we finally get (|8.9[) for all 
(t,x) G J x R". 

Proof of (|8 . 10|) : the proof of (|8.10[) uses the Weak Invariance Theorem for Time- 
Varying Systems (Theorem 12.2.2 QU pp. 446-448]). 

Take an arbitrary point (t, x) G [0, T) x R™ and y G W(i,x). Define the set-valued 
map Q from [i, T] to R n+P by 

Q(s)={(x,a), aef t ( s ,x)}, 

and consider the following optimal control problem: 

(x(s),a(s)) G (FL)(x(s)), a.e. s G [i,T], 
(x(t),a(t)) - (x,y), (8.12) 
(x(s),a(s)) G Q(s) for all s G [t,T], 

where (FL)(x) = cl(co({(f (x, u), — L(x, u)), u G J/})). Our objective is to apply the 
Weak Invariance Theorem for Time- Varying Systems to this problem. To do so, we 
need to check that the conditions required to apply this theorem are satisfied. First, 
as y G W(i,x), (x(f),a(i)) G Q{t), The condition (x(0),a(0)) G limsup, 5 ^ 0+ Q(s) di- 
rectlyfollows from assumption 3. Moreover, from the assumptions in the set-valued 
map FL possesses the required properties, and gph(Q) is closed as gph(Q) = gph(W-f) 
and Wf is outer semicontinuous by assumption 1. It remains to show the "inward- 
pointing" condition. Let (s,x, a) £ gph(Q) with s G (t, T) at which N sp mq\ (s, x, a) 
is nonempty. Take any vector (£*,v*,w*) £ -V gP h(Q) (s, x, a), we need to show that 

T+ min ((v*,w*),(f,L)) < 0, 

(fX)G(FL)(x) 

or 

C+ min (v*,f) + (w*,L) < 0. 

(fX)e(FL)(x) 

As already mentioned, gph(Q) = gph(Wf). Hence, iV gph (g)(s, x, a) = A^ gph(H / 1 . ) (s, x, a), 
and by definition of the coderivative (£*,v*) G D*W^(s, x, a)(— w*). Hence, from 
([8^]) . we get: 

f+ min (v*,f) + (-w*,L) < 0, 

(fX)e(FL)(x) X 

or 

C+ min (v*,f) + (w*,L) < 0, 

(fX)e(FL)(x) 

which is what was needed to show. 

The Weak Invariance Theorem for Time- Varying Systems therefore gives that there 
exists a function s G [t,T] — > (x(s),a(s)) solution to (|8.12p . Hence, for all s G 
[t,T], (x( S ),a(s)) eQ( S ), or 



a(s) G M/ t ( S ,x(s)). 
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Take s = T to get: 

a(T) G Wf(T, x(T)) = P. 

Now, we apply the Relaxation Theorem (Theorem 2.7.2 [TSJ p. 96]) to s £ [t,T] 
(x(s),a(s)). Hence, there exists a sequence u n (-) such that 

lim x(-;t,x, u n (-)) = x(-) and lim y + / -L(x(s;t,x,u„(-)),u„(s)) ds = a(-). 

In particular, for s = T, we get: 

limy - / L(x(s;t,x,u n (-)),u n (s)) ds = a(T). 

From above, we know that a(T) e P. Moreover, J t T L(x(s; i, x, u n (-)), u n (s)) ds = 
JO, T, x, u n (-))) G Y(t, x). Hence, 

y G cl(Y(t, x)) + P = V(t, x) + P - 7 t (t, x). 

This holds for all y £ W(t,x), hence 

W(t,x) C Vj.(t,x). 

From assumption 4 and recalling that V(T, x) = {0}, we finally get (18. 10)) for all 
(i,x) 6/xR".D 

Corollary 8.4. Let W be a set-valued map from I x R" to R p . Assume that W 
is an extremal element map. Then, under the assumptions of Theorem 18.3) we have: 

V(t,x) £l x R", W(i,x) = V(t,x). 



Proof. Let yi G F(t,x). From ()8.9|) . yi = w + di for some w G W(t, x) and 
di G P From (|8.10|) , w = y 2 + d 2 for some y 2 G V(t,x) and d 2 G P. Hence, 
yi = y 2 + di + d 2 . As V is an extremal element map, we must have di + d 2 = 0, 
which implies di = d 2 = 0, as P is pointed. Hence, yi £ W(t,x) which shows that 
V(t, x) C W(t, x). Inverting the role of V and W yields W(t, x) C V(t, x), and finally 
W(i,x) = V{t,x). □ 

Corollary 18 . 5 1 summarizes the findings of this paper. 

Corollary 8.5. The set-valued return map V is the unique generalized con- 
tingent and proximal solution for (MOC) such that the set-valued map Vf is outer 
semicontinuous. 

Proof. In we have shown that V is a generalized contingent solution for 
(MOC) and that the set-valued map Vf is outer semicontinuous. From Proposition 
18.21 it follows that V is also a generalized proximal solution for (MOC). Now, let W be 
a generalized proximal solution for (MOC) such that the set-valued map is outer 
semicontinuous. By definition of the generalized proximal solution, W is an extremal 
element map and satisfies the assumptions 2, 3, and 4 of Theorem l8.3l Moreover, Wf 
is outer semicontinuous. Hence, Corollarv l8. 41 applies and we get W = V. □ 
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